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^) image processing method and apparatus. 

(St) Images picked up by a plurality of cameras 
are corrected and projected on a single plane. 
The input images are converted to projected 
images on a reference projection plane in ac- 
cordance with reference parameters and cam- 
era parameters. Then images of an area of nx x 
ny whose center is an intersection B that is a 
perpendicular from a center of a lens of a m-th 
camera on the reference projection plane are 
extracted from the projected images. By ext- 
racting the area as described above, there is 
obtained an image which can be assumed that 
optical axes of each camera Intersects perpen- 
dicular to the reference projection plane and 
the optical axes are parallel to each other. 
Therefore, when an epipolar plane is construc- 
ted during an image interpolation process, cor- 
responding points form a straight line on the 
epipolar plane, thus it becomes easier to detect 
the corresponding points. 
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The present invention relates to an image processing method and apparatus and, nnore particularly, to an 
image processing method and apparatus capable of con-ecting image data. 

In a conventional method for finding corresponding points among a plurality of images, there is a method 
in which an epipolar plane is constructed and the corresponding points are detected on the epipolar plane. 

5 The epipolar plane is where an Image Is fonmed in such a manner that corresponding lines of a plurality of im- 
ages are extracted and these lines are arranged side by side. More specif ically, if a pixel of a n-th image at 
(i, j) is denoted by Pn(i. j), and a pixel in a m-th epipolar plane at (i, j) is denoted by Qm(i, j), then every pixel 
on the epipolar plane is expressed by Qm(i, j) = Pj(i, m). The method using the epipolar plane makes use of 
characteristics that, when a plurality of cameras, arranged at an equal interval, pick up an image of an object, 

10 the conresponding points in each picked-up image form a straight line on the epipolar plane. Thereby finding 
the conresponding points can be done by detecting the straight line. 

However, according to the conventional example, if image sensing surfaces (CCD, for instance) of all the 
cameras are not on a single plane, the corresponding points do not form a straight line on the epipolar plane. 
In that case, there arises a problem in which the process to detect the corresponding points becomes very 

15 complicated. 

Further, even though the image sensing surfaces of all the cameras are on a single plane, if the image 
sensing surfaces are rotated, the corresponding points do not form a straight line on the epipolar plane. Ac- 
cordingly, it becomes difficult to detect the corresponding points. 

The above-described problems can be solved by an-anging the plurality of cameras so that their image 

20 sensing surfaces are on a single plane and that their scanning directions are parallel to each other. However, 
it is very difficult, in general, to arrange the plurality of cameras so as to satisfy the aforesaid conditions. 

The present invention has been made in consideration of the above situation, and has as its object to pro- 
vide an image processing method and apparatus capable of conrecting and projecting images picked up by a 
plurality of cameras on a single plane. 

25 Embodiments of the present invention provide an image processing method and apparatus capable of ex- 

tracting desired portions of images projected on a single plane and outpulting it. 

Other embodiments of the present invention provide an image processing method and apparatus capable 
of extracting portions of images projected on the plane so that optical axes of cameras used for picking up the 
images of an object are parallel to each other. 

30 According to the present invention, the foregoing object is attained by providing an image processing ap- 

paratus comprising: input means for inputting an image picked up by an image sensing surface of a camera; 
calculation means for calculating a conversion parameter representing relationship between pixel positions on 
the image sensing surface and on a reference plane when the image on the image sensing surface of the cam- 
era is projected on the reference plane; forming means for forming an image of an area extracted from the 

35 reference plane by using data of pixels, which correspond to the pixels on the image of the area according to 
the conversion parameter, on the image sensing surface; and output means for outputting the image formed 
by the forming means. 

With the construction as described above, there are calculated the conversion parameters showing cor- 
respondence between pixel positions on the image sensing surface and on the reference plane when the im- 

40 age, picked up by a camera, on the image sensing surface is projected on the predetermined reference plane. 
By using the parameters, values at each pixel position in the portion extracted from the reference plane can 
be obtained from values at corresponding pixel positions on the image sensing surface. Accordingly, an image 
in an area on the reference plane can be formed. As described above, it is possible to con-ect an image on an 
image sensing surface of a camera and make an image on a predetemiined reference plane. 

45 According to an aspect of the present invention as described above, the calculation means calculates the 

conversion parameter representing correspondence between a coordinate system of the image sensing sur- 
face of the camera and a coordinate system of the reference plane on the basis of a reference parameter rep- 
resenting correspondence between a coordinate system of an object calculated in accordance with an Jmage 
taking result of the object and the coordinate system of the reference plane, and a camera parameter repre- ;^ 

50 senting correspondence between the coordinate system of the object and the coordinate system of the image 
sensing surface of the camera. 

With the construction as described above, the conversion parameters are calculated on the basis of the 
camera parameters and the reference parameters. The camera parameters are calculated based the image 
pick-up result of the object, and show correspondence between the coordinate system of the object and the 

55 coordinate system of the image sensing surface, and the reference parameters show correspondence between 
the coordinate system of the object and the coordinate system of the reference plane. When a plurality of cam- 
eras are used, for example, by calculating the conversion parameters for each camera, it is possible to project 
images of an object on the predetermined reference plane. 
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Further, preferably, the reference plane is a desired plane represented by the coordinate system of the 
object. Since a desired plane can be designated as the reference plane, it is possible to designates a plane 
which suits a plurality of camera. 

Further, preferably, the reference plane is a plane including the image sensing surface of a reference cam- 

5 era arranged at a desired image-taking position, further comprising reference parameter calculation means 
for calculating the reference parameter representing correspondence between the coordinate system of the 
image sensing surface of the reference camera and the coordinate system of the object on the basis of the 
image-taking result of the object by the reference camera before the calculation by the calculation means. By 
constructing as above, the reference plane is automatically set by setting a desired camera as a reference 

10 camera. 

Further, preferably, the area extracted from the reference plane by the forming means is an area whose 
center is an intersection of a perpendicular from a center of a lens on the reference plane. Accordingly, when 
images picked up by a plurality of cameras are corrected, optical axes of corrected images can be made parallel 
to each other. 

15 Further, preferably, the area extracted from the reference plane by the forming means is an area of one 

line necessary for forming an epipolar plane. 
' ■ Other aspects of the invention aim to make it easier to detect corresponding points by forming an epipolar 

plane from a plurality of images corrected and projected on a single plane and detecting the corresponding 

points as a straight line. 

' 20 Other aspects of the invention aim to make it possible to generate images seen from different viewpoints 

from those of input images by generating new lines in the epipolar plane by interpolation on the basis of the 
arrangement of the detected corresponding points on the epipolar plane and by using the lines. 

Still other aspects of the invention aim to make it possible to determine occurrence of occlusion in accor- 
dance with the arrangement of the corresponding points detected on the epipolar plane, thereby generate in- 
25 terpolated images of better precision. 

Other features and advantages of the present invention will be apparent from the following description tak- 
en in conjunction with the accompanying drawings, in which like reference characters designate the same or 
similar parts throughout the figures thereof. 

30 BRIEF DESCRIPTION OF THE DRAWINGS 

The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate 
embodiments of the invention and, together with the description, serve to explain the principles of the invention. 
Fig. 1 is a block diagram illustrating a brief configuration of an image processing apparatus according to 
35 an embodiment; 

Fig. 2 is a flowchart showing a processing sequence of the image processing apparatus according to the 
embodiment; 

Fig. 3 is a flowchart showing a process of calibration of a camera; 

Fig. 4 is a flowchart showing a second correcting process; 
-40 Fig. 5 is an explanatory view in which an image on an image sensing surface of a camera is projected on 

" a reference plane; 

Fig. 6 is a flowchart showing an image interpolation process; 

Fig. 7 is a flowchart showing an processing sequence of a corresponding point detecting process; 
Fig. 8 is an explanatory view in which a j-th epipolar plane is formed from each image (i = 1 to 4); 
45 Fig. 9 is a diagram explaining construction of data in the j-th epipolar plane; 

Fig. 10 is a flowchart showing an operational sequence of detecting a straight line at searching the cor- 
responding points; 

Fig. 11 shows the j-th epipolar plane in which the corresponding points are obtained by a corresponding 
point search; 

50 Fig 12 is a flowchart showing a processing sequence of an interpolation process of an image; 

Fig. 1 3 shows an epipolar plane when two images are generated between original images by interpolation; 
and 

Fig. 14 is an explanatory view of forming interpolation images from epipolar lines. 

55 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Preferred embodiment of the present invention will be described in detail in accordance with the accom- 
panying drawings. 
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In this embodiment, an image processing apparatus capable of generating images seen from viewpoints 
different from those of a plurality of cameras on the basis of images, picked up by the plurality of cameras, 
by interpolation will be described. In the Image processing apparatus according to the embodiment, Images 
from each camera are corrected so as to be on a single plane as a pre-processing step for the generating the 

5 interpolation images. 

Fig. 1 is a block diagram illustrating a brief configuration of the image processing apparatus according to 
the embodiment. In Fig. 1 , reference numeral 1 denotes an image input unit for inputting an image, and it con- 
sists of four cameras, 1a to 1d, in this embodiment. Reference numeral 2 denotes a first correction unit for 
performing confections, such as lens distortion correction, shading correction, color correction, and so on; 3, 

10 a matrix generator for generating a matrix used for correcting projection distortions caused by gaps in position 
and direction of the cameras; and 4, a second correction unit for correcting the projection distortions of an input 
image by using the matrix for correction generated by the matrix generator 3. 

Further, reference numeral 5 denotes a viewpoint detector for detecting a viewpoint of an observer. The 
viewpoint detector 5 detects the viewpoint by using a magnetic sensor, an ultrasonic sensor, and the like. It 

15 should be noted that the method for detecting the viewpoint is not limited to those described above, as far as 
the viewpoint can be detected. For example, the viewpoint can be detected by detecting the position and di- 
rection of the observer's head. Reference numeral 6 denotes an interpolation unit which searches correspond- 
ing points by using images corrected by the second correction unit 4 and generates interpolation images. A 
display 7 switches and displays images which are applied with interpolation or projection distortion correction 

20 in accordance with the viewpoint detected by the viewpoint detector 5. Note that these processes are controlled 
by a CPU which is not shown. 

Next, the flow of the entire processing of the image processing apparatus will be described. Fig. 2 is a 
flowchart showing a processing sequence of the image processing apparatus according to this embodiment. 
First at step S1, a CPU determines whether or not a camera has been calibrated. More specifically, it is 

25 determined whether or not a matrix for correcting projection distortions has been generated is determined. If 
a camera has not been calibrated yet, the matrix generator 3 calibrates the camera at step S2, and generates 
a matrix for correcting projection distortions. When the camera has been calibrated, the process proceeds to 
step S3, and an image is inputted from the image input unit 1 . Regarding the calibration of the camera at step 
S2, it will be explained later with reference to a flowchart in Fig. 3. 

30 The input image data is applied with first corrections, such as a lens distortion correction, shading correc- 

tion, color correction, and the like, by the first correction unit 2 at step S4. After this, the process moves to 
step S5 where the image data is applied with second corrections by using the matrix for correcting projection 
distortions, obtained during calibrating the camera, by the second correction unit 4. The second correction is 
to correct each image inputted by each camera of the image inputting unit 1 by using the matrix for correcting 

35 projection distortions, thereby convert each image into an image on a predetermined reference plane, namely 
an image on a single surface. The second correction at step S5 will be described later referring to a flowchart 
in Fig. 4. 

The process proceeds to step S6 where the viewpoint detector 5 detects a viewpoint of an observer. At 
step S7, the interpolation unit 6 generates interpolation images based on images corrected by the first and 
40 second correction units 2 and 4. In this case, images seen from viewpoints different from those of the inputted 
images picked up by the four cameras la to 1d are generated as the result of the interpolation process. The 
generation of images by the interpolation process will be described later with reference to Figs 6 to 14. 

At step S8, image data is selected in accordance with the viewpoint of the observer detected at step S6 
and sent to the display 7 where the image is displayed. In this embodiment, two proper images out of the input 
45 images and the interpolation images are treated as a pair of stereoscopic images, and the pair of images are 
displayed by switching alternatively at high speed. In this case, the observer can see a stereoscopic image 
by using shutter glasses which operate in synchronization with the alternating switching. If the images are po- 
larized in the directions differing from each other when they are displayed by switching alternatively at high 
speed in the display 7, the observer can see a stereoscopic image by using a polarization glasses which op- 
50 erate in synchronization with the alternating switching of the images for the right and left eyes. 
Next, the calibration of the camera at step S2 will be explained. 

Fig. 3 is a flowchart showing a calibration process of a camera performed at step S2. Thei projection dis- 
tortions are caused since each image is originally sensed on a different plane. Accordingly, in order to correct 
the distortions, images from N (N = 4, in this embodiment) input devices (cameras) need to be projected on a 
55 same plane. Here, one of the image sensing surface of the cameras is considered as a reference plane, and 
the camera whose image sensing surface is used as a reference plane is considered as a reference camera. 
Then, by projecting images obtained by the other cameras on the reference plane, images on a same plane 
can be obtained. 
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First at step S11 , the reference camera which is the desired camera (camera 1c. for example) of the image 
inputting unit 1 is set as the camera to be processed. At step S12, the reference camera picks up an image of 
an object. Then at step SI 3, the image obtained at step S12 is applied with the first correction. The correction 
here is the same as the one described at step S4, thus omitting the explanation. Next at step SI 4, camera 

5 parameters which represent relationship between a coordinate system of the object (world coordinate system) 
and a coordinate system of the image sensing surface of cameras (camera coordinate system) are calculated. 

As for calculating the camera parameters, a commonly known method for measuring a three dimensional 
image is used. More specifically, when the object is expressed in the world coordinate system (X. Y, Z) and 
the image sensing surface of each camera is expressed in the camera coordinate system ((xm. ym), m = 1 - 

10 4), the relationship between the world coordinate system and the camera coordinate system can be expressed 
by equation (1), 
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Cm is a m-th camera parameter. The camera parameters can be found by using a least squares method 
as far as coordinates of six points at least in the world coordinate system and corresponding coordinates in 
the camera coordinate system are known. At step S15, these camera parameters of the reference camera are 
decomposed in accordance with following equation (2), 
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A matrix Tm of the camera 1c which is obtained by decomposing in accordance with the equation (2) is 
stored as the standard parameter matrix Tb. It should be noted that any parameters having attached letter "b" 
have to do with the reference plane, hereinafter. The reference parameter matrix Tb is expressed by an equa- 
tion (3), 



50 



55 





b 1 2 


T 


T 


T 

b 21 


T 

b 22 


T 

^ b 23 




T^3, 


T 

b 32 


T 

^ o 33 




I 0 


0 


0 


1 



(3) 



If the camera parameters are known as described above, then the relationship between the world coor- 
dinate system and the camera coordinate system can be found. Accordingly, an equation for an image sensing 
surface of each camera in the world coordinate system is also found. The image sensing surface obtained as 
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above is defined as the reference projection plane (step S16). After calculating the reference projection plane 
of the reference camera, the first camera becomes a subject to be processed at step S17, then a matrix for 
correcting projection distortions is generated at step S18 and follow. 

At step S18, it is determined whether or not the camera which is the subject to be processed (referred as 
5 "camera to be processed", hereinafter) is the reference camera. If it is, since no correction is necessary, and 
the process moves to step S25. Whereas, if It is not, the process proceeds to step S1 9. At steps S19, S20 and 
S21, the same processes as at aforesaid steps S12, S13, and S14, respectively, are performed, then the proc- 
ess proceeds to step S22. 

At step S22, coordinates of the center of the lens of the camera to be processed is calculated. The coor- 
10 dinates of the lens center in the world coordinate system can be calculated by using the least square method 
by using characteristics that the normalized camera parameters, obtained by normalizing the camera parame- 
ters, and a straight line passing through each points in the image sensing surface intersect at the lens center. 
After the coordinate of the lens center is found, the process proceeds to step S23 where coordinates of inter- 
section of a perpendicular from the lens' center on the reference projection plane obtained at step S15 is cal- 
15 culated. The world coordinates of the intersection is denoted by (Um, Vm, Wm). Further, the coordinates of 
the intersection in the camera coordinate system of the reference camera Is denoted by (Xm, Ym), and stored. 
Note that this intersection is referred as "intersection B", hereinafter. In this case, relationship expressed by 
following equation (4) is established. 
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Then, the process moves to step S24 where a matrix for correcting projection distortions of the camera 
to be processed is calculated and stored. A matrix, Km. for correction of a m-th camera can be expressed as 

30 Km = Cm ■ Tb-^ After the obtained Km is stored, the process proceeds to step S25, and it is determined whether 
or not the camera to be processed Is the last camera (in this embodiment, four cameras are considered, thus 
the last camera is the fourth camera). If it is the last camera, the process goes back to a main routine (Fig. 2), 
whereas if it is not, the camera which is the subject to be processed is changed to the next one at step S26, 
and the process goes back to step S18. 

35 Next, the second correction process (the process at step S5) of correcting images inputted from each cam- 

era of the image input unit 1 by using the matrix for correcting projection distortions and of converting the im- 
ages into images on a predetermined reference plane, namely to images on a single plane will be explained. 

Fig. 4 is a flowchart showing a flow of the second correction process. First at step S31 , a variable m show- 
ing the ordinal number of camera Is initialized. Next at step S32, m Is incremented by 1 . The process proceeds 

40 to Step S33 where ym' = Ym - ny/2 and j = 0 are set, and at step S34, xm' = Xm - nx/2 and I = 0 are set. Here, 
Xm and Ym are x coordinate and y coordinate of the Intersection B. respectively, obtained at step S23. Further, 
nx and ny are numbers of pixels in the x direction and y direction, respectively, of an image to be processed. 
Furthermore, xm', ym', I, and j are variables. 

Fig. 5 is an explanatory view In which an Image of an Image sensing surface of a camera is projected on 

45 the reference projection plane. In Fig. 5, the coordinate system of the reference plane Is expressed by (X, Y), 
and the coordinate system of an input image Is expressed by (x, y). By using the aforesaid matrices for cor- 
rection, the input Image can be projected on the reference projection plane as a projected image. In this em- 
bodiment, a desired conrected image is an image obtained by extracting an area of hx x ny including the in- 
tersection B, found at step 523, as its center from the reference projection plane. By extracting the area as 

50 above, there can be obtained an image taken by such cameras that optical axes of them are assumed to in- 
tersect perpendicular to the reference projection plane and to be parallel to each other. Accordingly, upon con- 
structing an epipolar plane at the image interpolation process, the corresponding points forms a straight line 
in the epipolar plane, thereby it becomes easier to detect the corresponding points. 

Further, at steps S33 and S34, the variables are set to one pixel prior to start point S', which is the start 

55 point for extracting an area from the reference projection plane, in the x and y directions, so that the variables 
matches to coordinates of the start point S' after the processes at steps S35 and S36 are performed. 

At step S35, a pixel position of the image to be corrected is moved by one pixel at a time in the y direction 
by incrementing j and ym' by 1. Similarly, at step S36, a pixel position of the image to be corrected is moved 
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by one pixe! at a time in the x direction by incrementing I and xm' by 1 , The process proceeds to step S37 where 
a pixel in the input image to which the coordinates (xm\ ym') in the corrected image correspond is found. The 
coordinates (xm\ ym') can be obtained by solving following equation (5). 



10 



Hm « xm ^ 
Hm « ym 
Hm y 



I I 


^rr 1 7 


n 


c 


21 


22 


23 


2< 


31 


3? 


33 


34 



T 



bll 


T 

b 1 2 


T 


T 

bl4 


b21 


T 

^ b 22 


T 

^ b 23 


T 

b 24 


631 


T 

^6 32 


T 

^ D 33 


T 

b34 


0 


0 


0 


1 



y 




xm ' > 






ym • 






0 


J 


V 


1 J 



(5) 



15 



20 



25 



30 



35 



40 



45 



50 



55 



After the coordinates (xm. ym) are found, a pixel value of the input image at the pixel position is calculated, 
and the calculated value substitutes a value of a pixel whose pixel positron of the corrected image is expressed 
by (i, j). It should be noted that, at step S38. R(t) expresses an integer which is the closest to an arbitrary real 
value t. Further, Gm(i, j) and Fm(i, j) represent pixel values whose pixel positions are at (1. j) in the corrected 
image and the input image, respectively. After Gm(i, j) is calculated, the process moves to step S39 where it 
is determined whether or not i = nx. If i = nx, then the process proceeds to step S40, whereas if not, goes back 
to step S36. At step S40, whether or not j = ny is determined. If j = ny, then the process moves to step S41, 
whereas if not, goes back to step S34. 

By performing aforesaid processes at steps 34 to 40, the conrected image in the area of nx x ny centered 
by the intersection B in the reference plane is obtained. After the extraction of the area has been completed, 
the process proceeds to step S41 where it is determined whether or not m = N. If all the corrections on images 
from all the cameras are completed, namely m = N, the process goes back to the main routine in Fig. 2. On 
the other hand, if there is any unprocessed image, the process goes back to step S32. 

The corrected image generated as described above is a projected image on a single plane (reference 
plane), thus it can be considered that the optical axes of the cameras are parallel to each other. Therefore, if 
optical axes of each camera are not parallel to each other in practice, or if rotations, or the like, of each camera 
are slightly different from each other, it is possible to correct projection distortions of the images. 

Next, generation of interpolation images by the interpolation unit 6 will be described. 

Fig. 6 is a flowchart showing a flow of the interpolation process. First at step S51, corresponding points 
are searched. At step S52, an image seen from a viewpoint different from viewpoints of the input images is 
generated, then the process goes back to the main routine in Fig. 2. 

Fig. 7 is a flowchart showing an processing sequence of a corresponding point searching process at step 
S51. First at step S61, initialization is performed in order to extract the first raster of each image as a raster 
of interest. Next at step S62, the rasters of interest of each image are stored In a work memory, then a j-th 
epipolar plane (j-EPI) is virtually constructed. The j-th epipolar plane here is formed in such a manner that data 
of the j-th rasters are extracted from each image (1 = 1—4) and arranged, as shown in Ffg. 8. 

Fig. 9 Is a diagram explaining construction of data in the j-th epipolar plane. The epipolar plane is composed 
of four epipolar lines obtained by extracting corresponding rasters from the four images. 

The j-th epipolar plane is composed of a set of points, EPj(x, i), which satisfy following equation, 

EPj(x,i) = Ni(x,j) 

where Ni (x,j) denotes a x-th pixel value in j-th line of an l-th image (1 = 1-- 4), namely, a value of a pixel whose 
coordinates are (x, j) in the i-th image. For example, at EPj(x, 2) in Fig. 9, a value at (x, j) in the second image, 
namely N2(x, j) is stored. 

In a case where image sensing surfaces of input devices (cameras) are arranged at an equal interval in 
parallel to each other, the corresponding points (same point in each image) form a straight line on the epipolar 
plane. Therefore, the corresponding points can be determined by determining this straight line, further, inter- 
polation of images is to be performed on the deterrtiined straight line. Thus, at step S63, the straight line on 
which the corresponding points exist is determined on the j-th epipolar plane. Then the corresponding points 
are determined from the determined straight line and stored. 

A method for determining the straight line for searching the corresponding points on the epipolar plane is 
next described. Fig. 1 0 is a flowchart showing an processing sequence for determining a straight line for search- 
ing corresponding points. 

First at step S71, a preference order, n, is set to 1, and a raster of a pixel of interest, r, is set to 1. The 
preference order, n, shows relationship between objects overlaying in images. For example, the preference 
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order n = 1 represents an object in the image is in front (closer to the canneras), and will not be hidden behind 
other objects when a viewpoint is moved. Therefore, the corresponding points of the preference order 1 exist 
on all epipolar lines. Then, as the preference order decreases, namely the number of preference order, n, in- 
creases, there are more epipolar lines on which the corresponding points of the preference order do not exist. 
5 The process moves to step S72, and EPj(x, r) is considered as a pixel of interest, and all m's which satisfy 

the following equation (6) are calculated in ranges, m = k1 - k1 + k2, x = 1 nx. 

( X , r ) = X { ( + ^ < ^ - ^ 1 ) - ( >^ / ) }' < TH 2 

10 i = r + 1 

... (6) 

15 The value, k1, Is determined depending upon the input method, and when the cameras are arranged in 

parallel at equidistance from the object to pick up an image of an object, k1 is 1. The value, k2, is determined 
depending upon the distance between a camera and the object, and set to 20 (it is assumed that the viewpoint 
does not move more than 20 pixels) in this embodiment. Note, nx represents the number of pixels of an image 
in the main scanning direction. 

20 Note that m can be a real number, thus a value of x + m x (I - r) has to be rounded to the nearest whole 

number to determine the corresponding x coordinate. TH2 in the equation (6) is a threshold for finding the cor- 
responding points, and set to 1200 in this embodiment. The reason why it is set to 1200 is that the epipolar 
plane is composed of four rasters, thus differences are calculated three times. It is assumed that, if the dif- 
ference is about less than 20, colors are almost the same, the number 1200 is obtained from 3 x 20 x 20. 

25 If no specularity component exists in images inputted by an ideal input unit (corresponding points in each 

image have a same pixel value), TH2 = 0 is proper. However, in practice, pixel values of the corresponding 
points in each image differ, thus the difference is set to 20. Therefore, as more precise the input unit becomes, 
the difference becomes smaller. On the contrary, in an opposite case, the difference should be set large. 
The aforesaid method is described when a process is performed on each RBG pixel value, however, it can 

30 also be applied to a case where the value is converted into YIQ, HSI, or other color system, and the threshold 
can be set so as to suit the color systems. 

Further, when EPj(x + m x (i-r), i) does not exist (namely, x + m x (l-r) is outside of the range of the x axis 
(1 - nx)), the process continues assuming that the point corresponding to m does not exist. However, if EPj(x 
+ m X (i-r), i) has already been processed at step S72, the process continues assuming that EPj(x + m x (i-r), 

35 i) - EPj(x,r) is 0. 

Then the process proceeds to step S73 where the corresponding points of the preference order n are de- 
termined from the straight line of slope m obtained at step S72, and stored. When a plurality of corresponding 
points are found, all of such points are stored as corresponding points to the preference order n, for conve- 
nience. The pixels which are determined as corresponding points are considered to have been processed. It 

40 should be noted that, if a corresponding point overlaps another corresponding point of the same preference 
order, the point having the larger value of m (the one whose slope is gentler with respect to the epipolar lines) 
is preferred. This is because the movement of the point on images caused by the displacement of the viewpoint 
becomes greater as the object represented by the corresponding point is closer to the cameras (namely, the 
value of m becomes greater), thus the point having the larger value of m is preferred among a plurality of cor- 

45 responding points of the same preference order. 

In the process at step S73, when the corresponding points obtained from the straight line of slope m are 
processed (they are determined as corresponding points of higher preference order, and the corresponding 
points of a pixel currently interested is hidden behind the corresponding points of a higher preference order), 
the points are not considered as the corresponding points on the straight line of slope m. Then, number of 

50 unprocessed pixels, w, is set 

Then the process proceeds to step S74 where whether or not the number of unprocessed pixel is 0 is de- 
termined. If it is 0, the process is completed, and goes back to step S64 (Fig. 7), If it is not 0, the process moves 
to step S75 where it is determined whether or not the raster of interest, r, is the last raster, R, If the raster of 
interest, r, is the last raster, then the process proceeds to step S76, and the raster of interest, r, is set to 1 (first 

55 raster). On the other hand, if the raster of interest is not the last raster, then the process moves to step S77 
TA/here the value of the raster is incremented by 1 . Note that R at step S75 indicates the number of rasters (R 
= 4, in this embodiment) consisting the epipolar plane. 

Next, the process proceeds to step S78, and whether or not n, indicating the preference order, is equal to 
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N is determined. N indicates the complexity of the phenomenon in which objects hide other objects (occlusion). 
More specifically, if the value of N is large, a large number of objects are overlapped, whereas if the value of 
N is small, a small number of objects are overlapped. The number of N is set depending upon how in detail 
occlusion is expressed. Here, N is set to (R - 1) x 10, namely 30, as an empiric value. 

5 If it is determined that n is not equal to N at step S78, the process proceeds to step S79. Whereas, if it Is 

determined that n = N at step S78, the process is completed. At step S79, it is determined whether or not the 
number of unprocessed pixels, w, decreased comparing to the number of unprocessed pixel, u, in the former 
process. If the number of unprocessed pixels decreased, the process proceeds to step S80. At step S80, value 
of n in incremented by 1, and u is updated to the value of w, then the process goes back to step S72. On the 

10 other hand, if the number of unprocessed pixel did not decrease, then the process to detect a straight line on 
the j-th epipolar plane O'-^PI) is completed, and the process goes back to step S64 (Fig. 7). 

By performing the process in accordance with the aforesaid algorithm, corresponding points which are not 
found from two images can be determined. Further, the process deals with occlusion, and the like, thus the 
precision of searching corresponding points increases. 

15 At step S64 in Fig. 7, it is determined whether or not all rasters of input images are processed. If there is 

any unprocessed raster, the process moves to step S65 where value of j is incremented by 1 , then the process 
goes back to step S62. Whereas, if it is determined that alt rasters are processed, the corresponding point 
searching process is completed, and the process goes back to the flowchart in Fig. 6. Note, ny at step S64 is 
the total number of rasters of the input images. 

20 Next, the process at step S52 in Fig, 6, namely an image interpolation process, will be described. The image 

interpolation process is performed by using the corresponding points obtained at step S51. The processing 
sequence is described with reference to Figs. 11 to 14. 

Fig. 11 shows a j-th epipolar plane from which the aforesaid corresponding points are determined. In Fig. 
11, a1 and c1 represent corresponding points of the preference order 1, and b2 represents corresponding 

25 points of the preference order 2. A case where g images are interpolated between the input images at an equal 
interval will be described below. Note that, in this embodiment, p is chosen to be 2 in order to simplify the ex- 
planation. 

Fig. 12 is a flowchart showing an processing sequence of the image interpolation process. At step S90, a 
number of epipolar plane to be processed, j, is set to 1. Then at step S91, the preference order, n, is set to 1. 
30 At step S92, epipolar lines for interpolation images are inserted in the j-th epipolar plane. 

Fig. 13 shows an epipolar plane when two images are generated by interpolation between original images. 
Two epipolar lines are inserted between neighboring two epipolar lines on the j-th epipolar plane shown in Fig. 
11. 

Next at step S93, the corresponding points on the epipolar plane of the interpolation images are searched. 

35 Here, pixels on the straight line connecting the corresponding points on the epipolar plane of the input images 
are extracted as corresponding points. At step S94. whether or not a pixel assured as the corresponding point 
has been set as a corresponding point of the higher preference order is determined. If the pixel has not been 
set as the corresponding point of the higher preference order, the process proceeds to step S95, then a pixel 
value is set at the pixel as the corresponding point. The pixel value to be set at the corresponding point is an 

40 average of pixel values of corresponding points of the input images. 

Theorderof straight lines, obtained by connecting the corresponding points of the input images, to be used 
is the increasing order of the slope m when the preference order is same. Then, if the corresponding points 
indicate the same pixel, the value of the pixel is updated. Thereby, the corresponding points in a line having 
larger slope, m. is preferred. This is because the movement of the point on the images caused by the displace- 

45 ment of the viewpoint is greater when the object represented by the corresponding point is closer to the cameras 
(namely, the value of m is greater). Therefore, the points on the line having the larger value of m is preferred 
among a plurality of corresponding points of the same preference order, and thus occlusion can be dealt with. 

At step S95, values of pixels on the interpolated epipolar line and on the straight line which connects the 
corresponding points, as described above, are set to the average of values of pixels on the straight line con- 

50 necting corresponding points on the originals. When the corresponding points a1 and cl in Fig. 13 are consid- 
ered, the pixel values of points a and c on the straight line connecting the corresponding points are the average 
value of pixels represented by al and cl, respectively. 

Further, at a pixel (4. 9) in Fig. 13, a is set first, however, it is overwritten by the corresponding point c on 
the line having larger slope of m later. 

55 When the aforesaid processes are performed on alt the corresponding points having the preference order 

1 , n is incremented by 1 , and corresponding points having the preference order 2 start to be processed (steps 
S96 to S97). The processes to be applied to the corresponding points of the preference order 2 are the same 
as those performed at aforesaid steps S93 to S97. However, there are some pixels which have already inter- 
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polated when the corresponding points of the preference order 1 are processed, thus those interpolated pixels 
are not processed at step S94, 

This will be explained with reference to Fig. 13. A pixel (5, 8) is at the position which is to be interpolated 
with regard to corresponding points b2, however, it is already interpolated when the corresponding points cl 

5 of the preference order 1 is processed, thus the pixel (5, 8) is not processed. Further, since the corresponding 
points b2 do not exist on the first raster of the epipolar plane of the original (see Fig, 11), neither does it exist 
at the corresponding points b2 on the first to third epipolar lines in Fig. 13. Furthermore, in the example of Fig. 
13, occlusion occur at the points (4, 9) and (5, 8), and it is dealt with by the process described above. 

When the corresponding points having the preference order 2 are completely processed, it starts proc- 

10 easing conresponding points of the preference order 3. The aforesaid processes are repeated until all the cor- 
responding points of the last preference order are processed. 

After the corresponding points of the last preference order have been processed, the process proceeds 
from step S96 to step S98 where pixels which are not interpolated during the above-said processes are inter- 
polated with reference to neighboring pixels. As for the interpolation at this time, there are methods of using 

15 an average value of neighboring pixels and of using a value of the pixel closest to the pixel to be interpolated. 
At step S99 and S100, the aforesaid processes of steps S91 to S99 are performed on epipolar planes of 
j = i ^ ny. Accordingly, when the processes are performed on all the epipolar planes of j, the epipolar lines of 
j2, j3, j5, j6, jS, and j9 are generated by interpolation as shown in Fig. 1 3. In this embodiment, six interpolation 
images are obtained. Then at step S101, an interpolation image is generated from the interpolated epipolar 

20 lines generated as above. For example, an interpolation image 2 in Fig. 14 is generated by arranging the in- 
terpolated epipolar lines j2 (j = 1 - ny). Interpolation images 3, 5, 6, 8, and 9 can be generated in the same 
manner. 

By generating interpolation images from a plurality of input images by using the aforesaid configuration 
and method, corresponding points which could not be detected from two input images can be obtained, thus 

25 improving quality of interpolation. Further, conresponding points are found by using a plurality of input images, 
thus solving the occlusion problem, as described above. 

It should be noted that explanation about parallax in the vertical direction is omitted in the above embodi- 
ment, however, an image considering parallax in the vertical direction can be generated in such a manner that 
multi-viewpoint images picked up from image-taking viewpoints arranged at intersections of large meshes on 

30 a plane are stored and images are interpolated between the multi-viewpoint images in the right and left direc- 
tion, then in the vertical direction. Here, interpolation images in the vertical direction can be generated by ex- 
tracting a straight line formed by pixels arranged in the vertical direction and constructing an epipolar plane. 

Further, there are inputted images picked up by a plurality of cameras from different viewpoints, however, 
the input images can be picked up by a single camera by sliding its position so as to obtain equivalent images 

35 picked up by a plurality of cameras set at fixed positions. In this case, the direction of sliding of the camera 
and the x direction (main scanning direction) of the image sensing surface of the camera has to be parallel, 
or corresponding points will not form a straight line on an epipolar plane. However, the calibration described 
in the above embodiment can be applied to this case, too. Thus, it is apparent that it is possible to arrange 
corresponding points on a straight line. 

40 In a case where a plurality of images are taken by sliding the position of a single camera, first, images of 

objects are picked up at two positions. Then, a straight line connecting origins of camera coordinate systems 
at the two positions is defined as x axis of a reference projection plane, and y axis of an arbitrary camera co- 
ordinate system is defined as y axis of the reference projection plane. Since it is possible to find relationship 
between coordinate system of this reference projection plane and the world coordinate system, the relationship 

45 can be denoted as reference parameter matrix Tb. By using the thus obtained reference parameter matrix Tb. 
an interpolation image can be obtained from the equation (5), similarly to the above embodiment. 

Furthermore, in the above embodiment, one of the cameras is defined as the reference camera, and the 
reference projection plane is defined by the image sensing surface of the reference camera, however, the pres- 
ent invention is not limited to this. An arbitrary plane can be defined as the reference projection plane. In such 

50 a case, by obtaining the reference parameters by finding relationship between the world coordinate system 
and the coordinate system of the arbitrary plane, calibration can be performed in the same manner as in the 
aforesaid embodiment Note, since the equation of the reference projection plane in the world coordinate sys- 
tem is known, the relationship should be easily obtained. 

Further, in the above embodiment, a rectangular area is extracted from the reference projection plane be- 

55 fore the image interpolation processes, and the method for extracting the rectangular area is not limited to this. 
For example, it can be extracted by obtaining pixel values of one line necessary to generate an epipolar plane 
and projecting on the reference projection plane. More specifically, in the above embodiment, the j-th epipolar 
plane is constructed by extracting j-th rasters of images projected on the reference projection plane. Instead, 



10 



EP0 680 014 A2 



pixel values of coordinate points corresponding to only j-th rasters projected on the reference piane can be 
obtained, then constructing the j-th epipolar plane. For example, if y component of each pixel on the j-th raster 
is yj, then the j-th epipolar plane can be formed by using values of pixels expressed by (xm\ yj). 

According to the embodiment as described above, the input images on image sensing surfaces can be 
5 corrected to images on a single plane, thus search of corresponding points at the image interpolation process 
can be performed by finding a straight line on an epipolar plane. Accordingly, it becomes easier to search cor- 
responding points, thereby to generate interpolation images. 

The present invention can be applied to a system constituted by a plurality of devices, or to an apparatus 
comprising a single device. Furthermore, the invention is applicable also to a case where the object of the in- 
to vention is attained by supplying a program to a system or apparatus. 

According to the present invention as described above, images picked up by a plurality of cameras can 
be corrected and projected on a single plane. 

Further, according to another configuration of the present invention, it is possible to extract and output a 
desired image from images projected on the single plane. 
15 Furthermore, according to another configuration of the present invention, it is possible to extract a part of 

images projected on the single plane so that optical axes of the plurality of cameras picking up images are 
considered to be parallel to each other. 

Further, according to the present invention, an epipolar plane is formed from a plurality of images corrected 
to be images on the single plane, thereby corresponding points can be detected as a straight line. Accordingly, 
20 it becomes very easy to detect the corresponding points. 

Further, according to the present invention, new lines are generated by interpolation in an epipolar plane 
on the basis of the arrangement of the corresponding points detected on the epipolar plane, and a new image 
is generated by using the generated lines. Accordingly, it is possible to generate images seen from different 
viewpoints from those of the input images. 
25 Further, according to the present invention, occurrence of occlusion is detected from the arrangement of 

the detected corresponding points on an epipolar plane, thus more accurate interpolation images can be gen- 
erated. 

As many apparently widely different embodiments of the present invention can be made without departing 
from the spirit and scope thereof, it is to be understood that the invention is not limited to the specific embodi- 
30 ments thereof except as defined in the appended claims. 



Claims 

35 1. An image processing apparatus comprising: 

input means for inputting an image picked up by an image sensing surface of a camera; 
calculation means for calculating a conversion parameter representing relationship between pixel 
positions on the image sensing surface and on a reference plane when the image on the image sensing 
surface of the camera is projected on the reference plane; 
40 forming means for forming an image of an area extracted from the reference plane by using data 

of pixels, which correspond to the pixels on the image of the area according to the conversion parameter, 
on the image sensing surface; and 

output means for outputting the image formed by said forming means. 

45 2. The image processing apparatus according to claim 1 , wherein said calculation means calculates the con- 
version parameter representing correspondence between a coordinate system of the image sensing sur- 
face of the camera and a coordinate system of the reference plane on the basis of a reference parameter 
representing correspondence between a coordinate system of an object calculated in accordance with 
an image taking result of the object and the coordinate system of the reference plane, and a camera para- 

50 meter representing correspondence between the coordinate system of the object and the coordinate sys- 

tem of the image sensing surface of the camera. 

3. The image processing apparatus according to claim 2, wherein the reference plane is a desired plane rep- 
resented by the coordinate system of the object. 

55 

4. The image processing apparatus according to claim 2, wherein the reference plane is a plane including 
the image sensing surface of a reference camera arranged at a desired image-taking position, further 
comprising reference parameter calculation means for calculating the reference parameter representing 
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correspondence between the coordinate system of the image sensing surface of the reference camera 
and the coordinate system of the object on the basis of the image-taking result of the object by the refer- 
ence camera before the calculation by said calculation means. 

5 5. The image processing apparatus according to claim 1, wherein the area extracted from the reference 
plane by said forming means is an area whose center is an intersection of a perpendicular from a center 
of a lens on the reference plane. 

6. The image processing apparatus according to claim 1, wherein the area extracted from the reference 
10 plane by said forming means is an area of one line necessary for forming an epipolar plane. 

7. An image processing apparatus comprising: 

calculation means for calculating a conversion parameter for projecting an image picked up by a 

camera to a reference plane; 
15 forming means for fonming an epipolar plane by defining the processed picked-up image by using 

the conversion parameter as a projected image on the reference plane and by extracting a necessary 

area from the projected image; 

detecting means for detecting straight lines formed by corresponding points on the epipolar plane 

formed by said forming means; 
20 line generation means for generating an interpolated line between lines on the epipolar plane by 

arranging the corresponding points on the basis of the straight lines detected by said detecting means; 

and 

image generation means for generating a new image by using the interpolated line generated by 
said line generation means. 

25 

8. The image processing apparatus according to claim 7, wherein said detecting means detects the straight 
lines formed by the corresponding points which are not hidden by other corresponding points and defines 
then as straight lines of higher preference order, and a second straight line detected after excluding the 
straight lines of higher preference order is defined as a straight line of lower preference order, and wherein 

30 said line generation means uses pixel data of the straight lines of higher preference order with preference 

over the straight lines of lower preference order as pixel data to generate of interpolated line. 

9. The image processing apparatus according to claim 8, wherein, when a plurality of intersecting straight 
lines of same preference order are detected by said detecting means, a straight line whose slope is gentler 

35 with respect to the lines on the epipolar plane is dealt as the higher preference. 



10. An image processing method comprising: 

an input step of inputting an iniage picked up by an image sensing surface of a camera; 
calculation step of calculating a conversion parameter representing relationship between pixel pos- 
40 itions on the image sensing surface and on a reference plane when the image on the image sensing sur- 

face of the camera is projected on the reference plane; 

forming step of forming an image of an area extracted from the reference plane by using data of 
pixels, which con-espond to the pixels on the image of the area according to the conversion parameter, 
on the image sensing surface; and 
45 output step of outputting the image formed at said forming step. 

The image processing method according to claim 10, wherein, at said calculation step, the conversion 
parameter representing correspondence between a coordinate system of the image sensing surface of 
the camera and a coordinate system of the reference plane is calculated on the basis of a reference para- 
meter representing correspondence between a coordinate system of an object calculated in accordance 
with an image taking result of the object and the coordinate system of the reference plane, and a camera 
parameter representing correspondence between the coordinate system of the object and the coordinate 
system of the image sensing surface of the camera. 

55 12. The image processing method according to claim 11 , wherein the reference plane is a desired plane rep- 
resented by the coordinate system of the object. 

13. The image processing method according to claim 11, wherein the reference plane is a plane including the 

12 
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image sensing surface of a reference camera arranged at a desired image-taking position, further com- 
prising a reference parameter calculation step of calculating the reference parameter representing corre- 
spondence between the coordinate system of the image sensing surface of the reference camera and 
the coordinate system of the object on the basis of the image-taking result of the object by the reference 
5 camera before the calculation at said calculation step. 

14. The image processing method according to claim 1 0, wherein the area extracted from the reference plane 
at said forming step is an area whose center is an intersection of a perpendicular from a center of a lens 
on the reference plane. 

10 

15. The image processing method according to claim 1 0, wherein the area extracted from the reference plane 
at said forming step is an area of one line necessary for forming an epipolar plane. 

16. An image processing method comprising: 

15 a calculation step of calculating a conversion parameter for projecting an image picked up by a cam- 

era to a reference plane; 

a forming step of forming an epipolar plane by defining the processed picked-up image by using 
the conversion parameter as a projected image on the reference plane and by extracting a necessary 
area from the projected Image; 
20 a detecting step of detecting straight lines formed by corresponding points on the epipolar plane 

formed at said forming step; 

a line generation step of generating an interpolated line between lines on the epipolar plane by ar- 
ranging the corresponding points on the basis of the straight lines detected at said detecting step; and 
an Image generation step of generating a new image by using the interpolated line generated at 
25 said line generation step, 

17. The image processing method according to claim 16, wherein, at said detecting step, the straight lines 
formed by the corresponding points which are not hidden by other corresponding points are detected and 
they are defined as straight lines of higher preference order, and a second straight line detected after ex- 

30 eluding the straight lines of higher preference order is defined as a straight line of lower preference order, 

and wherein, at said line generation step, pixel data of the straight lines of higher preference order is used 
with preference over the straight line of lower preference order as pixel data to generate the interpolated 
line. 

35 18. The image processing method according to claim 17, wherein, when a plurality of intersecting straight 
lines of same preference order are detected at said detecting step, a straight line whose slope is gentler 
with respect to the lines on the epipolar plane is dealt as the higher preference. 

19. An apparatus for extracting a desired view point image from multiple input view point images derived from 
40 different camera positions, the apparatus including means for correcting at least one of said camera im- 
ages so as to bring the input image view points into a single plane and/or orientation. 

20. A method or apparatus having the features of any combination of the preceding claims. 

45 
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(54) Image processing method and apparatus 

(57) Images picked up by a plurality of canneras are 
corrected and projected on a single plane. The input im- 
ages are converted to projected images on a reference 
projection plane in accordance with reference parame- 
ters and camera parameters. Then images of an area 
of nx X ny whose center is an intersection B that is a 
perpendicular from a center of a lens of a m-th camera 
on the reference projection plane are extracted from the 
projected images. By extracting the area as described 
above, there is obtained an image which can be as- 
sumed that optical axes of each camera intersects per- 
pendicular to the reference projection plane and the op- 
tical axes are parallel to each other. Therefore, when an 
epipolar plane is constructed during an image interpo- 
lation process, corresponding points form a straight line 
on the epipolar plane, thus it becomes easier to detect 
the corresponding points. 
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CLAIMS INCURRING FEES 



The present European patent application comprised at the time of filing more than ten claims. 

□ 



Ait darns lees have been paid within the prescribed time limil. The present European search report has been 
drawn up tor ait claims. 



□ 



Only part of the claims tees have been paid witnin the prescribed time limil. The present European search 
report has been drawn up tor the first ten claims and for those daims for which daims fees have been paid, 

j namely daims: 

i 

I p— j No daims fees have been paid witNn the prescribed bme lirrut. The present European search report has been 
drawn up for the first ten claims. 



LACK OF UNITY OF INVENTION 



The Search Dtviston considers that the present European patent application does not comply with the requirement of unity ot 
invention ar>d relates to several inventions or groups of inventions, 

namely. 

1. Claims 1-6,10-15: 

An image processing apparatus/method aimed at projecting images picked up by 
input means on to a reference plane and to form an output image from an area 
extracted from said reference plane. 

2. Claims 7-9,16-18: 

An image processing apparatus/method aimed at detecting straight lines formed 
i in an epipolar plane and to generate interpolated lines in said epipolar planes, and 

I to generate from that information an output image or images. 

1 3. Claim 19: 

An apparatus for extracting a desired view point image from multiple input view 
point images. 

I 

j j — t All funner search tees have been pakJ witnin the fixed tme hmti. The present European search report has 
I been drawn up for all daims. 

1^ Only part of the further search fees have been paid witnin the fixec time timii. The present European search 
report has been drawn up tor those parts of the European patent appficaiion which relate to the inventions in 
respects o*. which search fees have been paid. 

namely ciatms: 1 '6, 1 0-1 5,7-9, 1 6-18 

j— j None of the further search tees has been paid within the fixed time limit. The present European search repoa 
f^s been dravm up for those parts of the European patent application which relate to the invention first 
mentioned in the claims. 

namely datms 
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